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Overview 


■  AF  Software  Engineering  Survey  2010 

■  Current  State  of  the  Field 

■  Perceived  Challenges 

■  Education  Needs 

■  Software  Professional  Development 
Program  (SPDP)  4.0 
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Software  Engineering  Survey 


■  AF  Software  Engineering  Survey  2010 

■  Initiated  by  AFIT/LS,  sponsored  by  AFMC/EN 

■  39  Question,  Web-based  Delivery 

■  Data  Collection:  8-28  Mar  10 

■  Purpose: 

■  What  is  the  state  of  software  engineering  in  the  AF? 

■  What  can  we  do  as  educators  to  help? 


(Requirements  Anyone?) 
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Background 


■  Found: 

■  Lists  of  software  engineering  challenges 

■  AF  policies  and  guidelines 

■  Industry  best  practices  and  education 

■  Challenging  to  identify: 

■  State  of  the  software  engineering  field  within  AF 

■  Characteristics  of  those  practitioners 

■  Cause  of  symptoms  (challenges) 

■  Effectiveness  of  current  education  system 
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Survey  Parameters 


Invited  ~24K  active  duty  &  civilian  employees 
to  take  a  voluntary  survey 

Invitees  were  limited  to: 

■  6  AF  Specialty  Codes  (AFSC’s) 

■  13  Civilian  Job  Series  Codes  (C JSC’s) 

Breadth  of  the  survey  was  limited  to: 

■  Air  Force  Material  Command  (AFMC) 

■  Air  Force  Space  Command  (AFSPC) 


Software  Career  Fields? 


Military 

■  33S  -  Communications  Officer 

■  61 S  -  Scientist 

■  62E  -  Developmental  Engineer 

■  63A  -  Acquisition  Manager 

■  3D0X2  -  Cyber  System  Ops 

■  3D0X4  -  Computer  System 
Programming 


Civilian 

■  0334  -  COMPUTER  SPECIALIST 

■  0340  -  PROGRAM  MANAGEMENT 

■  0343  -  MANAGEMENT  &  PROGRAM 
ANALYSIS 

■  0346  -  LOGISTICS  MANAGEMENT 

■  0801  -  GENERAL  ENGINEERING 

■  0850  -  ELECTRICAL  ENGINEERING 

■  0854  -  COMPUTER  ENGINEERING 

■  0855  -  ELECTRONICS  ENGINEERING 

■  0861  -AEROSPACE  ENGINEERING 

■  1101  -  GENERAL  BUSINESS  &  INDUSTRY 

■  1 51 5  -  OPERATIONS  RESEARCH 

■  1550 -COMPUTER  SCIENCE 

■  2210 -INFORMATION  TECHNOLOGY 
MANAGEMENT 
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Population  Size 


■  Difficult  to  determine  how  many  employees 
are  involved  in: 

■  Software  development 

■  The  management  of  software  development 


Assumption:  population  size  of  software 
engineering  practitioners  is  much  smaller 
than  invitee  list  A 

I  I 

Number  of  respondents:  858 
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Screening  Criteria 


■  Asked  invitee  to  fill  out  survey  under  any  of 
the  following  conditions: 

1.  Have  any  experience  developing  software  for 
the  Air  Force,  or 

2.  Have  any  experience  managing  a  project 
involving  the  development  of  software  for  the 
Air  Force,  or 

3.  Have  any  experience  serving  as  a  significant 
stakeholder  in  a  project  involving  the 
development  of  software  for  the  Air  Force 
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Sample  Size  by  Career  Field 


Number  of  Respondents  by  Career  Field 


Career  Field 

*  I 

Career  Field 

# 

0855  -  ELECTRONICS  ENGINEERING 

212 

0854  -  COMPUTER  ENGINEERING 

29 

0801  -  GENERAL  ENGINEERING 

83 

0861  -  AEROSPACE  ENGINEERING 

26 

62E  -  DEVELOPMENTAL  ENGINEER 

75 

3D0X4  -  COMPUTER  SYSTEM 

PROGRAMMING 

18 

1101  -  GENERAL  BUSINESS  & 

INDUSTRY 

72 

0850  -  ELECTRICAL  ENGINEERING 

12 

1550 -COMPUTER  SCIENCE 

62 

1515  -  OPERATIONS  RESEARCH 

10 

2210  -  INFORMATION  TECHNOLOGY 

MANAGEMENT 

62 

3D0X2  -  CYBER  SYSTEM  OPS 

10 

63A  -  ACQUISITION  MANAGER 

50 

OTHER 

8 

0346  -  LOGISTICS  MANAGEMENT 

47 

0340  -  PROGRAM  MANAGEMENT 

7 

33S  -  COMMUNICATIONS  OFFICER 

39 

61S- SCIENTIST 

4 

0343  -  MANAGEMENT  &  PROGRAM 

ANALYSIS 

31 

0334  -  COMPUTER  SPECIALIST 

1 
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Sample  Size  by  Career  Field 


%  Resi 


ALL  OTHERS 

28% 


0855- 

ELECTRONICS 

ENGINEERING 

25% 


63A-ACQ 

MANAGER 

6% 

2210 -INFO 
TECH  MGT 

7% 


1550 -COMPUTER 
SCIENCE 
7% 


0801  -  GENERAL 
ENGINEERING 
10% 


1101  -  GENERAL 
BUSINESS 

8% 


62E- 

DEVELOPMENTAL 

ENGINEERING 

9% 
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Distribution 
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#  Respondents  vs.  Year  Categories 

(Total  for  Year  Category) 

1  , 
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II  lull 

0-4  5-9  10-14  15-19  20-24  25-29  30-34  35-40 

(150)  (138)  (82)  (86)  (134)  (143)  (84)  (42) 


0334  -  COMPUTER  SPECIALIST 

■  0340  -  PROGRAM  MANAGEMENT 

■  0343  -  MNGT  &  PRGM  ANALYSIS 

■  0346  -  LOGISTICS  MANAGEMENT 

■  0801  -  GENERAL  ENGINEERING 

■  0850  -  ELECTRICAL  ENGINEERING 
0854  -  COMPUTER  ENGINEERING 
0855  -  ELECTRONICS  ENGINEERING 

■  0861  -  AEROSPACE  ENGINEERING 

■  1101  -  GEN  BUSINESS  &  INDUSTRY 
1515  -  OPERATIONS  RESEARCH 

■  1550  -  COMPUTER  SCIENCE 
2210 -INFO  TECH  MNGT 

■  33S- COMM  OFFICER 
3D0X2  -  CYBER  SYSTEM  OPS 

■  3D0X4  -  COMPUTER  SYSTEM  PROG 
61S-  SCIENTIST 

■  62E  -  DEVELOPMENTAL  ENGINEER 
63A-  ACQUISITION  MANAGER 
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Major  Survey  Themes 


■  Assess  the  Current  State  of: 

■  Experience 

■  Education  &  Certifications 

■  Software  Development/Acquisition  Challenges 

■  Identify  Education  Needs: 


■  Who  needs  it? 

■  What  do  they  need? 
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Years  Experience 


16 

14 

12 

10 

8 

6 

4 

2 

0 

0-4  5-9  10-14  15-19  20-24  25-29  30-34  35-40 

(150)  (138)  (82)  (86)  (134)  (143)  (84)  (42) 


Avg  Years  Project 
Management 

Avg  Years  Software 
Project  Management 

Avg  Years  Development 


Average  Years  of 
Experience  (Y-Axis) 
With  Respect  To  Years 
of  Service  (X-Axis) 


Experience 


Career  Field 

#  Resp 

%  Mgt  Exp 

%  Mgt  Qual 

%  Dev  Exp 

%  Dev  Qual 

0855  -  ELECTRONICS  ENGINEERING 

212 

59% 

67% 

80% 

71% 

0801  -  GENERAL  ENGINEERING 

83 

70% 

70% 

53% 

35% 

62E  -  DEVELOPMENTAL  ENGINEER 

75 

56% 

55% 

47% 

31% 

1101  -  GENERAL  BUSINESS  AND 

INDUSTRY 

72 

78% 

83% 

43% 

28% 

1550  -  COMPUTER  SCIENCE 

62 

66% 

79% 

90% 

81% 

2210  -  INFORMATION  TECHNOLOGY 

MANAGEMENT 

62 

68% 

79% 

82% 

68% 

63A  -  ACQUISITION  MANAGER 

50 

76% 

68% 

24% 

18% 

Percentage  of  Respondents  Indicating  a  Mid-to-High  Level  of  Experience  or  Qualification 


Compares: 

1 .  Management  to  Development  Experience/Qualification  Level 

2.  Experience  to  Qualification  Level 
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Served  As 
Program 
Manager 


Served  As 
Developer/ 
Engineer 


Distinct  #  Software  Projects 


Career  Field 

#  Resp 

Mean 

Std  Dev 

0340-  PROGRAM  MANAGEMENT 

7 

8.1 

6.4 

0854  -  COMPUTER  ENGINEERING 

29 

7.4 

10.4 

1101  -  GENERAL  BUSINESS  AND  INDUSTRY 

72 

5.5 

6.3 

1550 -COMPUTER  SCIENCE 

62 

4.7 

6.8 

0343  -  MANAGEMENT  &  PROGRAM  ANALYSIS 

31 

4.2 

6.3 

Across  All  Sampled  Career  Fields 

858 

3.6 

5.6 

Career  Field 

#  Resp 

Mean 

Std  Dev 

0854  -  COMPUTER  ENGINEERING 

29 

11.5 

12.9 

1550 -COMPUTER  SCIENCE 

62 

9.6 

12.8 

0850  -  ELECTRICAL  ENGINEERING 

12 

7.3 

8.6 

2210  -  INFO  TECH  MANAGEMENT 

62 

7.1 

10.6 

3D0X4  -  COMPUTER  SYSTEM  PROGRAMMING 

18 

6.1 

8.9 

Across  All  Sampled  Career  Fields 

858 

4.4 

7.4 
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Level  Experience  Per  Phase 


Level 

Requirements 

Arch/Design 

Construction 

Testing 

Sustainment 

None 

5% 

10% 

14% 

5% 

9% 

Slightly 

15% 

24% 

23% 

14% 

17% 

Moderately 

30% 

30% 

26% 

30% 

26% 

Experienced 

35% 

26% 

27% 

34% 

31% 

Extremely 

15% 

10% 

10% 

17% 

18% 

Total 

Responses 

853 

849 

847 

852 

851 

Percentage  of  Sample  Size  at  Various  Experience 
Levels  Per  Software  Lifecycle  Phase 
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Education  Sources 


DoD  Software  Acquisition 
Training  &  Education  Working 
Group  -  Competency  Framework 


SEI  -  Software  Engineering  Institute 


STSC  -  Software  Technology  Support  Center 


NDU  -  National  Defense  University 


SPDP  Education 


Not  Taken  SPDP  Courses 


116(14%)  724(84%)  19(2%) 


Inappropriate  curriculum 

Inappropriate  curriculum  for 

for  current  project 

89  current  project 

221 

Lack  of  interest 

40  Not  aware  of  SPDP 

71 

Not  previously  aware 

34  Little  perceived  value 

52 

Little  perceived  value 

34  Fully  supports 

32 

Courses  not  available  on 
my  schedule 

34 

Courses  on  desired  topics 

not  offered 

26 

Distance  learning  not 

appropriate  for  me 

16 

Yourself 

#  Indicated 

Lack  of  time 

205 

Supervisor 

#  Indicated 

Other  priorities  for  your  time 

313 

Didn’t  Answer 


Taken  SPDP  Courses 
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External  Certification 


Certification 

#  Certified 

(All  Career  Fields) 

Project  Management  Professional  (PMP) 

38 

IEEE  Certified  Software  Development 
Professional  (CSDP) 

6 

IEEE  Certified  Software  Development 

Associate  (CSDA) 

4 

Engineering  License  w/  Software  Engineering 
Specialization 

10 

ASQ  Certified  Software  Quality  Engineer 

4 

INCOSE  Certified  Systems  Engineering 
Professional 

5 

At  Best,  Only  67  (7.8%)  of  Respondents  Indicate 
Having  External  Certifications 
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Organizational  Attributes 


Organizational  Attribute  (#  Respondents) 

Not  Exhibited  / 
Poorly 

Satisfactorily 
/  Exemplary 

Establishing  accurate  performance,  cost,  and 
schedule  baselines  (821) 

63% 

37% 

Understanding  policy  and  guidance  for  software  38%  62% 

acquisition  and  development  (810) 

Identifying  the  appropriate  composition  of  team  37%  63% 

members  (814) 

Establishing  and  adhering  to  a  defined  37%  63% 

architecture  (814) 

Adequately  addressing  system  integrity/security  35%  65% 

(811) 

Project  leadership  has  appropriate  39%  61% 

understanding  of  how  software  impacts  overall 
program  (811) 
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Organizational  Attributes 


Organizational  Attribute  (#  Respondents) 

Not  Exhibited  / 
Poorly 

Satisfactorily 
/  Exemplary 

Choosing  appropriate  software  lifecycle  model 
(811) 

39% 

61% 

Choosing  appropriate  software  methodology 
(808) 

32% 

68% 

Choosing  appropriate  implementation  language 
(810) 

31% 

69% 

Educating  stakeholders  as  to  their  role  in 
software  acquisition/development  (811) 

60% 

40% 

Capturing  lessons  learned  (811) 

56% 

44% 

Disseminate  lessons  learned  to  external 
organizations  (810) 

74% 

26% 
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Observations 


■  “#1  Reason”  why  AF  software  projects  fail  to 
meet  cost,  schedule,  or  performance 
(741  “free  responses”) 


■  Simple  Word  Search 

■  Requirements:  490 

■  Management:  132 

■  User:  49 

■  Experience/expertise:  45 

■  Design:  39 

■  Integration:  24 

■  Architect:  1 1 


22 


Observations  (cont.) 


■  Requirements:  change,  creep,  underfunded,  under¬ 
analyzed,  poor  quality,  inaccurate 

■  Management:  unrealistic  expectations,  poor  quality, 
unclear  direction,  failing  to  listen  to  engineers,  lack  of 
understanding  software  complexity,  inappropriately 
handled  of  issues,  wrong  focus,  lack  of  oversight, 
micromanagement,  lack  of  technical  background, 
over-promises 

■  User:  not  involved,  indecisive, 
unrealistic  expectations, 
doesn’t  understand  own 
requirements 
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Education 


■  How  education  may  address  the 
aforementioned  challenges...  (679  responses) 


■  Simple  Word  Search 


■  Management/Managers/Leadership:  217 

■  Engineer:  91 

■  Experience:  54 

■  Class:  32 

■  Practice:  26 

■  Learn:  32 

■  Example:  17 
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Education  (cont.) 


■  Management/Managers/Leadership: 
education  on  software  project  management, 
technical  concepts,  software  estimation, 
expectation  management,  team  composition 

■  Engineers:  education  on  requirements 
gathering/analysis,  interpersonal  skills, 
software  “engineering”  rather  than 
programming  education,  testing 

■  Practice/Learn/Example/Experience:  best 
practices,  lessons  learned,  examples 
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Most  Desired  Topics 


Response 

Personally 

a 

Boss 

a 

Coworker 

a 

Team  Management 

205 

-1 

401 

2 

159 

-1 

Testing 

261 

1 

126 

-1 

315 

2 

Stakeholder  Expectation 

205 

-1 

334 

2 

171 

-1 

Architecture/Design 

358 

2 

170 

-1 

360 

2 

Requirements  Engineering 

401 

3 

331 

2 

419 

3 

o  =  standard  deviation 


Respondents  Were  Asked  to  Choose  Up  to  5  Topics 
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Policy  -  Guidance  -  Resources 


AF  Specific  (#  Respondents) 

Unaware 

Know/Apply  It 

N/A 

USAF  Weapon  Systems  Software 
Management  Guidebook  (813) 

59% 

36% 

5% 

Guidelines  for  Successful  Acquisition  and 
Management  of  Software-Intensive  Systems 
(GSAM)  (807) 

68% 

29% 

2% 

AFI  63-1201  Life  Cycle  Systems  Engineering, 
Attachment  8  Software  Engineering  (804) 

39% 

59% 

1% 

AFI  63-101  Acquisition  &  Sustainment  Life 
Cycle  Management,  Section  3.84  Software 
Engineering  (797) 

36% 

63% 

2% 

Space  &  Missile  System  Center  Software 
Acquisition  Handbook  (800) 

72% 

22% 

6% 

AF  System  Engineering  Assessment  Model 
(AFSEAM)  (804) 

50% 

48% 

2% 
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Policy  -  Guidance  -  Resources 


Industry  /  Academia  (#  Respondents) 

Unaware 

Know/Apply  It 

N/A 

Software  Lifecycle  Processes  (IEEE  12207) 
(810) 

37% 

61% 

2% 

Guide  to  the  Software  Engineering  Body  of 
Knowledge  (SWEBOK)  (804) 

67% 

31% 

2% 

Software  Engineering-Software 

Measurement  Process  (ISO/IEC  15939) 

(797) 

55% 

44% 

2% 

Capability  Maturity  Model  Integration  (806) 

17% 

81% 

2% 

Six  Sigma  Process  Improvement  (804) 

7% 

91% 

1% 

ISO  9000  Quality  Management  Systems 
(803) 

10% 

89% 

1% 
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Conclusions 

■  Significant  problem  areas 

■  Requirements 

■  Management 

■  Very  little  sharing  among  software 
development  organizations 

■  Practitioners  are  sensitive  to  time  for  PCE 

■  Unaware  of  resources  out  there 

■  Missing  the  AF-specific  piece  on  policy  &  best 
practices 

■  Very  little  external  certification 
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SPDP  4.0 


■  Partition  management  and  engineering 
concerns  across  2  tracks 

■  Shorter  courses  over  distance  learning 

■  4  weeks  3  weeks  w / 18-24  hrs  of  coursework  per 
course 

■  2-8  hour  courses  for  special  topics 

■  Industry  Standards  ->  AF  Implementation  of 
Industry  Standards;  Best  Practices 
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SPDP  4.0  (cont.) 


■  New  learning  objectives 

■  Knowledge-focused  Assessment-focused 

■  Working  towards  AF  qualification  for 
software  engineering  personnel 

■  AF  Software  Project  Manager  Certificate 

■  AF  Software  Engineer  Certificate 
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Backbone  Guidance 


DEPARTMENT  OF  THE  AIR  FORCE 
Software  Technology  Support  Center 


Guidelines  for  Successful 
Acquisition  and  Management  of 
Software-Intensive  Systems: 
Weapon  Systems 
Command  and  Control  Systems 
Management  Information  Systems 


Condensed  Version 
February  2003 


Guide  to  the  Software  Engineering  Body  of 
Knowledge 


2004  Version 


SWEBOK® 


A  project  of  the  IEEE  Computer  Society 
Professional  Practices  Committee 


v  /EEE  -  'fli'W  F>nvnrj 


*SWEBi"JK  j:i  .in  eJtidiii  \enriee  ousU  uftlie  IFEF 


32 


Program  Overview 


SW  Engineering  (Development)  Track 


I 


SPDP 
Certified 


SWE  399 
Current  Topics 


Current  Know 


SWEBOK-focus 

l 


SWE  499 
Current  Topics 


e-focus 
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■  Mr.  Mike  Nicol,  ASC/EN 

■  Mr.  Dominick  Tuccillo,  AFMC/EN 

■  Mr.  Ernesto  Gonzalez,  SAF/AQRE 
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Summary 


■  AF  Software  Engineering  Survey  2010 

■  SPDP  4.0 


For  more  information: 
mark.reith@afit.edu 

SPDP@afit.edu 


U.S.AIR  FORCE 
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